• Tuesday, September 3, 2024

    Amazon S3 can be used as a strongly consistent event store with the help of the new conditional writes feature. Conditional writes enable optimistic concurrency, which is necessary for managing data updates in distributed systems. This article provides a step-by-step guide on how to implement this strategy. It also goes over structured naming conventions and conflict resolution.

    Hi Impact
  • Tuesday, September 3, 2024

    Amazon S3 now supports conditional writes that can check for the existence of an object before creating it. This helps developers more easily prevent applications from overwriting any existing objects when uploading data. Conditional writes can be used to simplify how distributed applications with multiple clients concurrently update data in parallel across shared datasets. Developers no longer need to build any client-side consensus mechanisms to coordinate updates or use additional API requests to check for the presence of an object before uploading data. The feature is available at no additional charge in all AWS regions.

  • Thursday, May 23, 2024

    S3, Amazon's object storage service, is missing features like compare-and-swap (CAS), multi-region buckets, and object appends. S3 Express One Zone, S3's faster alternative, lacks many standard S3 features and has high storage costs. As S3 continues to not have these modern features, developers will have to either build workarounds or move on to an S3 alternative.

    Hi Impact
  • Monday, June 3, 2024

    Amazon S3 has some aspects that can lead to security vulnerabilities. This article goes through some of these, such as the ability to perform certain actions without authentication, alternative ways to retrieve object keys, incomplete multipart uploads that can be deleted but not downloaded, and the potential for accidental public exposure of buckets through misconfigurations.

  • Friday, August 23, 2024

    Elastic Block Store (EBS) at AWS has evolved from simple shared drives to a massive, distributed SSD system delivering over 140 trillion operations daily. This post shares key lessons learned over the years, emphasizing the importance of incremental improvements, comprehensive instrumentation, and the power of constraints to drive innovation. It also discusses how a team tackled performance challenges by addressing the entire system stack, from the hypervisor to the network, and ultimately by building their own SSDs optimized for EBS.

  • Thursday, May 23, 2024

    While S3 is undoubtedly a feat of engineering, its feature set is falling behind its competitors. S3 doesn't have a compare-and-swap operation, something every other competitor has, and it also lacks multi-region buckets and object appends. Engineers wanting any of these features have to either abandon S3 or build around these gaps.

  • Friday, April 19, 2024

    Supabase Storage now supports the S3 protocol with a new API and newly supported multipart uploads with the S3 protocol.

  • Monday, August 12, 2024

    This blog post shows off a novel web dev architecture that removes the need for a separate database. It uses in-memory data storage, periodic snapshots, and transaction logs for high availability. The author details how the Raft Consensus Protocol can be integrated to achieve 99.999% availability, making it suitable even for demanding enterprise clients.

  • Thursday, June 20, 2024

    SSDs have a fast read and write throughput, but also suffer from high latency during write operations. To mitigate this issue, database systems use techniques like group commits and asynchronous processing. Enterprise-grade SSDs with capacitor-backed write caches can alleviate the latency problem, but at a higher cost.

    Hi Impact
  • Thursday, June 6, 2024

    This blog post explores different architectures for building real-time collaborative applications with a central server. It discusses the challenges of server-side rebasing and solutions. The post also delves into the complexities of handling text and list editing in collaborative apps and recommends using optimistic local updates to improve user experience.

  • Tuesday, August 13, 2024

    The "Read After Write" problem arises when a service attempts to read data immediately after writing it, but the changes may not be reflected in all nodes of the database yet. This is a common problem in distributed systems. This article goes over different database replication strategies, in terms of latency and data consistency. It suggests potential solutions, such as retrying reads, reading from the primary node, and accepting occasional inconsistencies depending on the application's requirements.

    Hi Impact
  • Tuesday, September 10, 2024

    Misconfigured AWS S3 buckets can be hacked through various means, such as examining HTTP responses, using search engines, and bruteforcing common keywords. Developers can test for misconfigurations by testing for list, read, write, and download permissions, examining Access Control Lists (ACLs), and checking for missing file type restrictions and S3 versioning. Knowing how to do this is important to actually secure your S3 buckets properly against unknown attackers.